-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
added intent evaluator #56
base: main
Are you sure you want to change the base?
Conversation
WalkthroughThe recent updates introduce a new class Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 6
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- athina/evals/llm/intent/evaluator.py (1 hunks)
- athina/evals/llm/intent/prompt.py (1 hunks)
Files skipped from review due to trivial changes (1)
- athina/evals/llm/intent/prompt.py
Additional Context Used
Ruff (4)
athina/evals/llm/intent/evaluator.py (4)
2-2:
typing.Tuple
imported but unused
2-2:
typing.Optional
imported but unused
4-4:
athina.interfaces.result.DatapointFieldAnnotation
imported but unused
74-74: Local variable
label
is assigned to but never used
Additional comments not posted (7)
athina/evals/llm/intent/evaluator.py (7)
17-19
: LGTM!
21-23
: LGTM!
25-27
: LGTM!
29-31
: LGTM!
33-35
: LGTM!
37-40
: LGTM!
46-47
: LGTM!
def __init__(self, **kwargs): | ||
self.model_name = "HUGGINGFACE_META_LLAMA_3_70B" | ||
super().__init__(**kwargs, system_message_template=INTENT_EVAL_PROMPT_CONCISE_SYSTEM, | ||
user_message_template=INTENT_EVAL_PROMPT_CONCISE_USER,) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Make the model name configurable.
- self.model_name = "HUGGINGFACE_META_LLAMA_3_70B"
+ self.model_name = kwargs.get("model_name", "HUGGINGFACE_META_LLAMA_3_70B")
Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
def __init__(self, **kwargs): | |
self.model_name = "HUGGINGFACE_META_LLAMA_3_70B" | |
super().__init__(**kwargs, system_message_template=INTENT_EVAL_PROMPT_CONCISE_SYSTEM, | |
user_message_template=INTENT_EVAL_PROMPT_CONCISE_USER,) | |
def __init__(self, **kwargs): | |
self.model_name = kwargs.get("model_name", "HUGGINGFACE_META_LLAMA_3_70B") | |
super().__init__(**kwargs, system_message_template=INTENT_EVAL_PROMPT_CONCISE_SYSTEM, | |
user_message_template=INTENT_EVAL_PROMPT_CONCISE_USER,) |
@property | ||
def examples(self): | ||
pass |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Mark the examples
property as abstract or implement it.
- def examples(self):
- pass
+ @property
+ def examples(self):
+ raise NotImplementedError("Subclasses should implement this method.")
Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
@property | |
def examples(self): | |
pass | |
@property | |
def examples(self): | |
raise NotImplementedError("Subclasses should implement this method.") |
@@ -0,0 +1,92 @@ | |||
import time | |||
from typing import List, Tuple, Optional |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove unused imports.
- from typing import List, Tuple, Optional
+ from typing import List
Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
from typing import List, Tuple, Optional | |
from typing import List |
import time | ||
from typing import List, Tuple, Optional | ||
|
||
from athina.interfaces.result import EvalResult, EvalResultMetric, DatapointFieldAnnotation |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove unused imports.
- from athina.interfaces.result import EvalResult, EvalResultMetric, DatapointFieldAnnotation
+ from athina.interfaces.result import EvalResult, EvalResultMetric
Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
from athina.interfaces.result import EvalResult, EvalResultMetric, DatapointFieldAnnotation | |
from athina.interfaces.result import EvalResult, EvalResultMetric |
failure = self.is_failure(result) | ||
passed_value = 1 - float(failure) | ||
metrics.append(EvalResultMetric(id=MetricType.PASSED.value, value=passed_value)) | ||
label: str = self.label |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove unused variable label
.
- label: str = self.label
Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
label: str = self.label |
malicious_keywords = ["malicious", "illegal", "harm", "harmful", "unlawful", "hurt", "pain", "hate"] | ||
for keyword in malicious_keywords: | ||
if keyword.lower() in chat_completion_response_json["result"].lower(): | ||
self.label = "malicious" | ||
self.label = "normal" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix the label assignment logic.
- for keyword in malicious_keywords:
- if keyword.lower() in chat_completion_response_json["result"].lower():
- self.label = "malicious"
- self.label = "normal"
+ self.label = "normal"
+ for keyword in malicious_keywords:
+ if keyword.lower() in chat_completion_response_json["result"].lower():
+ self.label = "malicious"
+ break
Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
malicious_keywords = ["malicious", "illegal", "harm", "harmful", "unlawful", "hurt", "pain", "hate"] | |
for keyword in malicious_keywords: | |
if keyword.lower() in chat_completion_response_json["result"].lower(): | |
self.label = "malicious" | |
self.label = "normal" | |
malicious_keywords = ["malicious", "illegal", "harm", "harmful", "unlawful", "hurt", "pain", "hate"] | |
self.label = "normal" | |
for keyword in malicious_keywords: | |
if keyword.lower() in chat_completion_response_json["result"].lower(): | |
self.label = "malicious" | |
break |
Summary by CodeRabbit
New Features
Improvements
Usability